RELOPS-2402: fleetbench PSU/firmware-throttle detector (hardware fleet) - #1263

Reverts 3cb9415 and 12978d1. Fleetbench is an unrelated feature that happens to
sit on this branch; it has its own branch (RELOPS-2402-fleetbench) and its own open
PR (#1263), so carrying it here would duplicate that review.

data/os/Windows.yaml and win_scheduled_tasks/files/maintainsystem-hw.ps1 are touched
by both features, so this is a per-hunk split rather than a file-level one. The
RELOPS-2487 changes to both are preserved; only the fleetbench hunks are removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ed-WIM pipeline (#1297)

* Add win116424h2hwbake role for wim-packer image bake

Trimmed copy of win116424h2hw that omits the four deploy-time/
machine-specific/hardware profiles (windows_worker_runner, microsoft_kms,
nuc_bios, nuc_management) so the stable catalog can be baked into a golden
install.wim on a reference VM, Sysprep-generalized, and captured. The omitted
profiles run at first boot on the real NUC via the full win116424h2hw role.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Add Hiera data for win116424h2hwbake role

Mirrors win116424h2hw.yaml without generic_worker.client_id (worker
registration is excluded from the bake role).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* win116424h2hwbake: drop scheduled_tasks from the bake; note Chrome freshness

- Exclude scheduled_tasks from the bake role. Those are runtime/operational tasks
  (maintain_system startup loop, self_redeploy_check -> PXE redeploy, gw_exe_check,
  task-user logon) that could fire during the bake or on the generalized image's
  first boot before the deploy-time run. The full win116424h2hw role registers them
  at deploy time (same rationale as excluding windows_worker_runner).
- TODO comment on google_chrome: baked Chrome can go stale between bake and deploy;
  ensure it's refreshed before the first worker-runner start.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* win116424h2hwbake: drop nsclient + admin from the bake (deploy-time)

Remove hardware_observability (win_nsclient monitoring agent; looks up marlin_pw)
and windows_datacenter_administrator (local admin password from win_adminpw) from
the bake role. Both are deployment-time identity/config tied to the datacenter and
must not be baked into a generalized image; the full win116424h2hw role applies them
at first boot. Consequence: the bake role now references NO Vault secrets, so the
bake needs only an empty placeholder vault.yaml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* win116424h2hwbake: drop google_chrome + chocolatey from the bake (deploy-time)

Chrome is the only chocolatey consumer in the role. Baking it (a) requires the
chocolatey provider, which is not functional in the bake's single puppet pass
(provider suitability is decided at run start, before choco installs), and (b)
goes stale between bake and deploy. Install Chrome at worker deployment instead:
keeps it current and removes chocolatey from the bake entirely. Everything else
(7-Zip, MozillaBuild, Git, etc.) installs directly from the assets blob, not choco.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* win116424h2hwbake: drop ssh from the bake (deploy-time)

OpenSSH server + the winaudit SSH key on the Administrator account are datacenter
access/identity, set up at deploy. They also can't apply cleanly on a generalized
bake image: the Administrator profile dir doesn't exist, and enabling sshd in a
single pass races the OpenSSH install. Defer the whole ssh profile to deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* bake fixes: tooltool PAT via env, empty-PAT guard, idempotent wsearch disable

Support the wim-packer image bake (no D: secrets drive, single puppet pass):

- facts_win_github.ps1: fall back to $env:custom_win_github_pat when D:\secrets\pat.txt
  is absent, so the bake can supply the GitHub token as a build-scoped env var (from
  the pipeline / GHA GITHUB_TOKEN) instead of a secrets drive. D: still takes priority
  for datacenter/MDC1 deploys; never written to disk in the bake.
- download_tooltool.ps1.epp: only send the GitHub Authorization/API headers when a PAT
  is present. An EMPTY Bearer on a public raw.githubusercontent.com URL returns 404
  (not 401), which broke the bake; also fixes production re-applies with a momentarily
  empty fact. With a real PAT, behavior is unchanged.
- wsearch/disable.ps1: make it idempotent (guard the takeown/rename on the indexer
  still existing). Previously it failed on every re-apply after the first — puppet
  re-applies on a schedule.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* win_uninstall_appx_packages: check presence before removing / logging

The removal loop logged "removing AppX match: <X>" for every key in the list
unconditionally - so a WIM already debloated in the bake looked like it was
removing ~48 apps on every run (misleading; the real removals are silent inside
try{}). Snapshot provisioned (DISM, survives AppXSvc-disabled) + installed once,
then per app: remove if actually present, otherwise log "not installed, skipping".
Makes the log reflect reality and avoids spawning removal jobs for absent apps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* maintainsystem-hw: disable refresh-rate reimage for the RELOPS-2487 canary

CompareConfigBasic's refresh-rate check reimages when CurrentRefreshRate != 60.
nuc13-160 is headless/on a KVM and reports 1, so a SUCCESSFUL deploy immediately
Set-PXE'd into a reimage loop. Comment out the reimage (keep logging the value)
so the canary stays up to troubleshoot. TODO: restore / make headless-aware
before master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* maintainsystem-hw: drift-check against the canary's dev branch, not main

CompareConfigBasic downloads pools.yml from main and reimages if the node's ronin
hash / image dir don't match. The RELOPS-2487 canary deploys from the nuc-wim-pipeline
branch (hash + image live there), so checking against main's relops1213 (edef633 /
older image) was a permanent false "config mismatch" -> Set-PXE loop right after a
good deploy. Point the drift-check URL at the dev branch. TEMP: revert to main
before master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* maintainsystem-hw: restore strict refresh-rate reimage check

Re-enable the bug-1910123 check (reimage when CurrentRefreshRate != 60) that was
disabled for the RELOPS-2487 canary. A NUC that reports refresh rate 1 has fallen
back to the Microsoft Basic Display Adapter (the real Intel GPU driver isn't loaded)
- a bad environment for our workers, so reimaging it is correct. The fix for the
pre-baked-WIM path is to inject the Intel drivers into the WIM (win-hw-wim
drivers.inject) so the node comes up at 60, not to soften this check.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* maintainsystem-hw: suppress refresh-rate reimage while troubleshooting (RELOPS-2487)

The pre-baked node still reports refresh rate 1 (Intel GPU driver not active) even
after baking the NUC13 driver pack, so the strict check reimage-loops the canary.
Comment out Set-PXE (keep logging the observed rate) to stop the loop while we
diagnose why the Intel display driver isn't loading. Restore once it reports 60.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* maintainsystem-hw: gate on generic-vs-Intel display adapter, not refresh rate

Refresh rate was a fragile proxy: these NUCs run through a Raritan KVM whose EDID
advertises 64 Hz (healthy != 60), while a driverless node reports the sentinel 1.
Check the real signal instead: is the display still on the generic Microsoft Basic
Display Adapter (no Intel GPU driver) vs a real Intel GPU. This is KVM-session-
independent (the driver binds whether or not a Raritan session is open, and it's
closed during test runs). Set-PXE stays commented while we get the Intel graphics
driver into the bake; restore it once the real GPU is in use.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* maintainsystem-hw: re-enable reimage on generic-adapter (GPU now baked in)

The golden WIM now bakes in the Intel graphics driver (proven on nuc13-160: iGPU
binds Intel Iris Xe @ 60 Hz), so a node on the generic Microsoft Basic Display
Adapter is a genuine failure. Restore Set-PXE in the adapter check (suppressed
during driver troubleshooting). Keeps the adapter-based detection (not the fragile
refresh-rate proxy).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* uninstall_appx_packages: drop 'not installed, skipping' per-key log

On a pre-baked (already debloated) image nearly every key on the ~48-app list is
absent, so this DEBUG line was ~48 lines of pure noise per run. Skip silently when a
package isn't present; keep logging actual removals.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* RELOPS-2402: fleetbench PSU/firmware-throttle detector (hardware fleet)

Split out of PR #1235 (combined RELOPS-2396 branch) into fleetbench-only.

Per NUC13 hardware node, before worker-runner starts:
- win_fleetbench module installs the version-pinned collector to C:\fleetbench
  plus per-hardware baselines, wired via the hardware_observability profile.
- maintainsystem-hw.ps1 runs `fleetbench cpu --mode quick --duration 900s --json`
  once post-bootstrap then at most once per 72h; 900s self-warms the node so
  PSU/thermal throttling surfaces (a short cold-boot run can false-pass).
- Evaluates GOOD/BAD/MARGINAL/UNKNOWN vs the locked per-hardware baseline, plus
  drift vs the node's first recorded run.
- NSClient++ checks `fleetbench` and `fleetbench_variance` surface verdict +
  metrics to Marlin (Icinga2/Grafana).

Hardware-only: gated to the datacenter maintain-system path + hardware_observability.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 2aac40e)

* RELOPS-2402: skip gw_exe_check watchdog while fleetbench benchmark runs

Invoke-FleetbenchCheck runs `fleetbench cpu --duration 900s` (~15 min) before
worker-runner starts, so generic-worker is intentionally not up during the run.
The hourly gw_exe_check task can fire mid-benchmark once uptime passes its
15-minute grace period, see no generic-worker process, and escalate to
reboot/PXE reimage.

maintainsystem-hw.ps1: set MOZ_FLEETBENCH_RUNNING (run's UTC start time) at
Machine scope around the benchmark call, cleared in a finally so it is always
removed even on error.

gw_exe_check.ps1: read the marker live from the registry
([Environment]::GetEnvironmentVariable(..,'Machine'), not the possibly-stale
process env block) and skip the check while it is fresh. The marker is a
timestamp with a 30-min staleness cap, so a crash/reboot mid-run cannot
silently disable the watchdog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 74e8909)

* RELOPS-2487: don't bake the AppXSvc disable; skip the AppX wait when there is no work

disable_appxsvc writes AppXSvc Start=4 AND registers the at-startup task
\Hardening\Hard-Disable-AppXSvc. During a WIM bake both are captured into the
golden install.wim, which silently defeats the ref/ref-alpha pool exemption
added for bug 2013985: that exemption only makes puppet DECLINE to disable the
service at deploy time, it never re-enables it. Deployed ref-alpha nodes
therefore came up with AppXSvc disabled, the HEVC/AV1/VP9/WebMedia extensions
stayed provisioned but never registered for the per-task user, and Firefox fell
back to 'ffvpx video decoder (RDD remote)' instead of the WMF hardware decoders
-- failing mochitest-media-mda-gpu (HEVC unsupported, VP9/AV1 not hardware
decoded, hevc/mkv playback timeouts).

Skip disable_appxsvc when custom_win_role ends in 'bake' so the image ships
AppXSvc Manual and each pool's deploy-time run makes the call, which is what the
exemption always assumed.

Leaving AppXSvc enabled in the image has a cost: wsappx can then run, so
Wait-AppxIdle in win_uninstall_appx_packages.ps1 is no longer instantly-idle and
would block up to 600s on every deploy for zero work on an image the bake already
debloated. Add a cheap pre-check (two enumerations, no waiting) that skips both
the wait and the removal pass when nothing on the removal list is present. The
app-key list is hoisted to script scope so the pre-check and the removal share
one list.

Deliberately NOT done as a puppet-level reorder of disable_appxsvc ahead of the
removal: most of the fleet still runs the MDT-built win11-24H2-NUC-01-16-2025
image and relies on the deploy-time removal, which a reorder would silently stop.
The pre-check is inert on those nodes because keys are present.

The shortcut is gated on the provisioned enumeration having actually succeeded --
that view is DISM-level and does not need AppXSvc, so a failure there means we
cannot see the image and must not mistake it for "already debloated".

* RELOPS-2487: stage codecs at bake, register at deploy; stop PXE-looping on a generalized name

Reverses the approach in b64eb0b. That commit stopped the bake disabling AppXSvc so
the image would ship it Manual. It fixed the codecs but changed sysprep's behaviour:
the new WIM gets a full generalize, so the first-boot specialize pass regenerates a
random WIN-xxxxxxxx into ActiveComputerName. maintainsystem-hw then looked the node up
under that name, missed, and Set-PXE'd -- re-imaging into a fresh random name, forever.

The bake never needed to change. The HEVC/AV1/VP9/WebMedia packages are PROVISIONED
into the image, which is DISM-level and needs no AppXSvc; only the per-user
registration at first task_* logon does. So: stage at bake, register at deploy.

- disable_services.pp: the bake disables AppXSvc again (restoring the known-good
  sysprep path). ref/ref-alpha now get an ACTIVE re-enable rather than an exemption --
  merely declining to disable is useless on a pre-baked WIM, which arrives with
  Start=4 and the \Hardening\Hard-Disable-AppXSvc startup task already in place.

- win_disable_services::enable_appxsvc (new): removes the hardening task FIRST (it runs
  at startup and would otherwise re-disable the service on the next boot), then sets
  Start=3/Manual -- matching stock Windows and the working MDT reference nodes.

- maintainsystem-hw.ps1 CompareConfigBasic: resolve the node from the PERSISTENT
  ComputerName instead of $env:COMPUTERNAME (ActiveComputerName), which is the value
  specialize corrupts. Where the two disagree, take an ordinary reboot to activate the
  pending rename -- bounded to 2 attempts so a name that never activates degrades into
  a loud log rather than an endless cycle. And never Set-PXE on a name still matching
  ^win-: that means "identity not established yet", not "wrong node", and re-imaging
  over it is what made the loop unbreakable.

Observed on nuc13-158: ComputerName=NUC13-158 but ActiveComputerName=WIN-D81J5HC82S0,
with Tcpip Hostname/NV Hostname both correct -- so OS-deploy's offline rename worked and
only the active name was clobbered.

* RELOPS-2487: stop Write-Log corrupting Get-AppxSnapshot's return value

win_uninstall_appx_packages.ps1's Write-Log emitted the message with Write-Output,
which puts it on the PIPELINE. Any Write-Log call inside a function that returns a
value therefore appends the log line to that function's return.

That went live with the win11-24h2-hw-20260820-235936 WIM. The bake disables AppXSvc
again, so at deploy Get-AppxPackage throws, Get-AppxSnapshot's catch logs a WARN, and
the function returns Object[] { '<warning text>', <hashtable> }. Binding that to the
[hashtable] $Snapshot parameter threw:

  Cannot convert the "System.Object[]" value ... to type "System.Collections.Hashtable"

so the script exited 1, puppet exited 6, and bootstrap's exit-6 branch ran Set-PXE.
Every node re-imaged onto the same image and failed the same way - a hard loop, hit on
all five canary nodes (t-nuc12-002/003, nuc13-158/074/115).

Write-Host instead: still rendered to stdout for Puppet's logoutput and still captured
by the transcript, but never on the pipeline. The duplicate UserInteractive Write-Host
block is now redundant and removed.

Note the same Write-Output-in-Write-Log pattern is still present in win_enable_appxsvc.ps1
and win_disable_appxsvc.ps1. Neither logs from inside a value-returning function today,
so neither is broken, but both are one edit away from this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: point self_redeploy_check's drift check at the canary branch too

maintainsystem-hw.ps1's CompareConfigBasic was repointed at the nuc-wim-pipeline
pools.yml back in August, but self_redeploy_check.ps1 - a SECOND, independent drift
checker installed on every datacenter Windows node - still read main.

The canary pools carry their hash and image on the dev branch; main holds only the
`dev:` trigger and a stale hash (a22e7ac). So self_redeploy_check compared the node's
GITHASH against main's and logged "Git Hash MISMATCH!" on every run, while
CompareConfigBasic on the same node logged "Git Hash Match: f1eeb1f" a minute later.

Observed on all five canary nodes: a mismatch sets $SETPXE, and with no active task
that path is Set-PXEPendingFlag -Clear + Restart-Computer, so the nodes were rebooting
roughly hourly (t-nuc12-002 18:53, 19:06, 19:49, 20:00; nuc13-158 18:31, 19:32, 20:01).
Had a task been running instead, the other branch calls Set-PXE and re-images the node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: enable VBS + HVCI in puppet so baked nodes match the MDT image

A node deployed from the pre-baked golden WIM came up with Virtualization-Based
Security entirely off, while the MDT-built production image runs VBS with both
Credential Guard and HVCI. HVCI runs the kernel under a hypervisor and taxes
syscalls, memory management and I/O, so the two populations were not measuring
the same machine - baked nodes looked faster than the fleet they represent.

Measured on nuc13-006 (production) against nuc13-115 (baked); the delta was
exactly two registry values:

  HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard
    EnableVirtualizationBasedSecurity = 1
  HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity
    Enabled = 1

Nothing else was needed, which is worth recording because both are easy to get
wrong:

  - Credential Guard needs no key of its own. Production shows
    SecurityServicesRunning 1,2 with no CredentialGuard scenario key and
    LsaCfgFlags unset - it is default-on for Windows 11 Enterprise once VBS is up.
  - No Hyper-V optional feature is required. Production has every
    Microsoft-Hyper-V-* feature Disabled yet reports HypervisorPresent=True;
    Windows loads the hypervisor for VBS on its own.

New win_device_guard module, driven from hiera via a device_guard profile, wired
into the bake role and the two canary roles. Both nodes already report
AvailableSecurityProperties 1-8 and Secure Boot on, so the hardware is ready.

Takes effect on the next boot, which the deploy gets for free (bootstrap reboots
after the puppet run). In the bake VM the values are written but VBS will not
activate - nested virtualisation is not exposed to the packer guest - which is
expected: the WIM carries the setting and it activates on real hardware.

Not added to win116424h2hw: production already has this from its image, so
puppet-enforcing it there is a separate, fleet-wide decision.

puppet-lint clean (5.1.1, repo flag set).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: install Intel Graphics Software so baked nodes get IntelGraphicsSoftwareService

Production MDT nodes run IntelGraphicsSoftwareService; nodes built from the pre-baked
WIM do not. It is not a driver service - it ships inside the AppUp.IntelArcSoftware
MSIX. Under DCH the graphics *software* is a separate Store-delivered package that
Windows Update fetches as a driver companion app, and our images have WU disabled by
design, so the MU-catalog driver cab (INF only) can never supply it.

Intel's full installer does carry it, as
Resources/Extras/IntelGraphicsSoftware_26.18.2353.2_Release.exe - confirmed by scanning
gfx_win_101.7088.exe. Note that win_packages::drivers::intel_gfx passes --noExtras,
which is exactly the flag that skips that folder; this class does not pass it.

Package uploaded to hardwareimaging resources/drivers/gfx_win_101.7088.exe
(771,672,256 bytes, sha256 6c2cd36a5c68cf3aa8403073e5ed4592b8ee4ba5190f5776e483d974ead5a2f8).

Stage at bake, install at deploy - and it has to be that way, because the blob is
Entra-only: an anonymous GET to hardwareimaging returns 409, while the deploy-time
mirror roninpuppetassets returns 200. The bake build host holds a managed identity and
can azcopy the installer in; a deployed NUC has no Azure identity and cannot. So this
class never downloads. It is idempotent and self-selecting:

  bake   - service absent + installer staged locally -> runs it into the image
  deploy - service already present from the WIM      -> no-op, starts it if stopped

Missing installer is a WARN, not a failure, so a node on an older WIM is not bricked by
a catalog error. The script also reports whether the package ended up provisioned
image-level, since a per-user-only install would be stripped by sysprep /generalize.

STILL REQUIRED (worker-images, not this repo): prepare-base-vhdx.ps1 must stage the
installer to C:\bake\extras\ during the bake, the same route the driver cabs take.
Until that lands this class is a no-op everywhere.

puppet-lint clean; script AST_OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: keep Intel Graphics Software NUC13-only, matching production

Production is not uniform. The NUC13 pool (win11-64-24h2-hw) has
IntelGraphicsSoftwareService; the NUC12 reference pool (win11-64-24h2-hw-ref) does not -
measured on t-nuc12-004/005/006, all three report the service absent. The goal is to
replicate production, so ref-alpha (NUC12) must not have it either.

One golden WIM serves both platforms and the bake provisions the MSIX image-wide, so the
NUC12 pools cannot simply decline to install it: by the time puppet runs it is already in
the image. They need an ACTIVE removal, the same shape as
win_disable_services::enable_appxsvc actively undoing the baked AppXSvc disable.

So win_intel_graphics_software gains $ensure:
  present (default, NUC13 + bake) - install if the service is absent, else start it
  absent  (NUC12 roles)           - remove the provisioned and installed package

The exec guard is the mirror image of $ensure, so both directions are idempotent.

$ensure is a profile PARAMETER rather than a plain hiera lookup because the discriminator
is the role, and the Windows hiera hierarchy has no role level: win_hiera.yaml keys its
per-worker level on custom_win_gw_workertype, and that fact is EMPTY on these hardware
pools - HKLM\SOFTWARE\Mozilla\ronin_puppet\workerType is unset on nuc13-115, t-nuc12-002
and t-nuc12-004 alike, so that level never matches anything. win116424h2hwrefalpha
therefore declares the profile with ensure => 'absent'; NUC13 roles take the default.

Removal is best-effort and logs each step. Get-AppxPackage enumeration is expected to fail
wherever AppXSvc is disabled; the provisioned removal is the one that matters on a freshly
deployed node with no extra user profiles.

puppet-lint clean; script AST_OK.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: make win_packages::drivers::intel_gfx work, and target 101.7088

Groundwork for moving the NUC12 reference pool off the 31.0.101.3729 its MDT image
shipped. ref and ref-alpha are currently a whole major driver version apart
(31.0.101.3729 vs 32.0.101.7085), which makes the reference pool not comparable to the
pool it is the reference for.

This class was commented out of every role and could not have worked if enabled:

  1. it staged the installer to <systemdrive>\intel\gfx.exe but executed
     C:\Windows\Temp\gfx.exe - a path nothing ever wrote to;
  2. its guard read $facts['custom_display_adpater'] (sic), a fact defined nowhere in
     this repo, so it was always undef and the guard always true;
  3. the file resource fetched a ~750 MB installer on every run whether or not the driver
     already matched.

Now: one guarded exec that downloads, installs and cleans up, skipped entirely when
Win32_VideoController already reports the target version. A node already on target
transfers nothing.

--noExtras is kept and is load-bearing: it skips Resources/Extras, where
IntelGraphicsSoftware_<ver>_Release.exe lives. Production's NUC12 reference pool does not
have IntelGraphicsSoftwareService and must not gain it, so this class installs the driver
and nothing else. The NUC13 pools get the software separately via
win_intel_graphics_software, which runs the same installer without that flag.

hiera windows.driver.gfx.version 101.4255 -> 101.7088.

NOT YET ENABLED ANYWHERE. roles_profiles::profiles::intel_drivers is still commented out
in win116424h2hw, win116424h2hwref and win116424h2hwrefalpha, and it is BLOCKED on
hosting: the class pulls from the public roninpuppetassets mirror, which has only
gfx_win_101.4255.exe - 101.7085 and 101.7088 both 404 there. Deployed NUCs have no Azure
identity and cannot read our Entra-only hardwareimaging account, so the installer has to
be on the public mirror before ref can move.

puppet-lint clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: install Intel Graphics Software at deploy, not at bake

IntelGraphicsSoftwareService is present on MDT production NUC13 nodes and absent on
every pre-baked one. The install was wired into the bake role and has never once
worked; this moves it to the deploy-time roles, where it is proven to work.

Two independent reasons the bake cannot do it:

  1. Intel's installer returns rc=1008 in the packer guest. On real NUC13 hardware
     the same installer, driven by this same script, returns rc=1001 and leaves
     IntelGraphicsSoftwareService Automatic/Running (measured on nuc13-115,
     2026-09-01). The GPU-less Hyper-V build guest is the obvious difference.
  2. Even on success it would not survive capture. Intel lays the
     AppUp.IntelArcSoftware MSIX down as a per-user install, not a provisioned one -
     the script's own check reported `provisioned packages matching: 0` - and
     sysprep /generalize strips per-user AppX from the image.

So the WIM now ships the INSTALLER rather than the installed software:

  - installer_path C:\bake\extras\gfx_win_*.exe -> C:\extras\gfx_win_*.exe.
    worker-images prepare-base-vhdx stages extras.files there instead of under
    C:\bake, which sysprep-generalize.ps1 deletes wholesale before capture (it holds
    the SYSTEM puppet helper embedding the build GitHub token, so that scrub stays).
    That deletion is the entire reason deployed nodes found no installer and silently
    took the best-effort skip branch.
  - win116424h2hwbake drops the profile, so the bake stops burning time on an install
    that cannot succeed.

No change to the deploy-time contract: perf-debug and the other NUC13 roles keep
ensure => present, ref-alpha keeps ensure => absent, and the exec is still guarded on
the service, so a node that already has it does nothing.

Also reworded the provisioned-package log line. Zero provisioned packages is the
EXPECTED result of a per-user install and is not a fault; the service is the success
signal. The old wording warned about sysprep stripping the WIM, which is meaningless
in the deploy-time context this now runs in.

Requires a re-bake: the current golden WIM has no C:\extras.

Ref: RELOPS-2487

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* RELOPS-2487: defer the Intel Graphics Software install to maintainsystem

Measured on the 2026-09-08 deploy of win11-24h2-hw-20260908-172915: Intel's installer
takes 2m46s-3m03s (nuc13-115/074/158) and ran inside the deploy puppet apply, roughly
doubling its ~2.5-2.9 min. It is most of the NUC13-vs-NUC12 spread on that deploy -
16-18 min against 12-13 min for a pool that never runs it.

Nothing at deploy time needs the service, so blocking the node from reporting ready on
it buys nothing. win_intel_graphics_software now drops a marker holding the installer
path, and the first maintainsystem run after deploy consumes it.

  present -> marker only, no exec. maintainsystem-hw.ps1 installs it.
  absent  -> unchanged, still inline. Removal is a fast no-op on a node that never had
             the service, and a NUC12 pool must not be left carrying it even briefly.

The marker doubles as the ensure discriminator: only NUC13 roles get one, so
maintainsystem needs no role logic of its own. Its content is the installer path, so
hiera stays the single source of truth. maintainsystem deletes it BEFORE the attempt -
a failed or crashed install must not retry three minutes on every boot. The task is
boot-triggered and runs as SYSTEM, which the installer needs.

Deploy-time change only, no re-bake: the WIM already carries C:\extras.

KNOWN TRADE, worth stating for a perf pool: the node now reports ready ~3 min sooner,
but generic-worker can claim a task while the installer is still running on the first
boot after a re-image. Treat the first task after a re-image as suspect, or wait for
"Intel Graphics Software done" in the MaintainSystem event log before trusting numbers.

Ref: RELOPS-2487

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "RELOPS-2487: defer the Intel Graphics Software install to maintainsystem"

This reverts commit ccee773.

The deferral does not actually save the three minutes, it relocates them. The work is
irreducible either way; all that changes is whether the node is blocked during it or
busy during it.

  inline      puppet blocks ~3 min -> registers later -> the post-puppet reboot has
              already happened, so the first task runs on a settled machine
  deferred    registers ~3 min sooner -> maintainsystem installs at boot, concurrently
              with worker-runner starting, so the first task after a re-image can land
              on a machine mid-install

For a general CI pool the first framing is the right one and the deferral is a real win.
perf-debug and ref-alpha exist to produce trustworthy numbers, and there it buys close to
nothing: "3 min not ready" becomes "3 min ready-but-unreliable", plus a standing
requirement to remember to discard the first task after every re-image. Bad trade on the
two pools currently being used to chase a 2.8% difference.

Removing the 3 min for real needs the AppUp.IntelArcSoftware MSIX provisioned into the
WIM so nothing installs at deploy. That is blocked on sourcing, not on hardware -
Add-AppxProvisionedPackage is image-level and needs no GPU, which is how the media
codecs already get baked - but no standalone .msixbundle exists to provision: a prod node
keeps only the unpacked signed staged directory, winget msstore returns nothing, Intel's
site blocks scripted access, and gfx_win_101.4255.exe does not contain it. The one
unexplored route is the copy the MDT share holds, which is where production got
26.26.2459.0.

Back to a312287, which is verified working on all five canary nodes: perf-debug
GFXSVC Running/Automatic, ref-alpha correctly ABSENT, ERRDEV 0 on both.

Ref: RELOPS-2487

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Remove RELOPS-2402 fleetbench from this branch (it ships in #1263)

Reverts 3cb9415 and 12978d1. Fleetbench is an unrelated feature that happens to
sit on this branch; it has its own branch (RELOPS-2402-fleetbench) and its own open
PR (#1263), so carrying it here would duplicate that review.

data/os/Windows.yaml and win_scheduled_tasks/files/maintainsystem-hw.ps1 are touched
by both features, so this is a per-hunk split rather than a file-level one. The
RELOPS-2487 changes to both are preserved; only the fleetbench hunks are removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Point the drift checks back at main (drop the canary override)

Both CompareConfigBasic (maintainsystem-hw.ps1) and CompareConfig
(self_redeploy_check.ps1) were temporarily pointed at the nuc-wim-pipeline branch's
pools.yml so the canary pools, whose hash+image live on that branch, would not see a
permanent false "config mismatch" and Set-PXE themselves in a loop. Both carried an
explicit "REVERT this URL to main before master" marker; this is that revert.

self_redeploy_check.ps1 is now identical to master and drops out of this changeset
entirely - the canary URL was its only change.

NOTE for anyone re-imaging the canary: keep those pools pinned to 459102e until this
lands on master. The branch TIP now drift-checks against main, which does not carry the
canary hash/image, so deploying the tip to a canary pool would reintroduce the loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>